Enhanced Build Process

The Enhanced Build Process is an all new data load engine for FastStats systems.

Enable the Enhanced Build Process by setting it is the build process on the Additional options dialog within the System Configuration:

Overview

The Standard Build Process uses a number of 'sequential file access' techniques that allow relatively low power systems to build large FastStats systems without requiring a large amount of memory. However by taking advantage of the amount of memory available on modern machines the new Enhanced Build Process uses a different approach to build systems more quickly.

Build Process Methodology

The Standard Build Process consists of two main phases:

1. Sorting: Sort the input data files into the correct order required for the FastStats System Join

2. Load: Building the sorted input files into the FastStats compressed .dat files

The Enhanced Build Process reverses these phases:

1. Compression: Compress the unsorted files into FastStats compressed .dat files

2. Re-index: Sort the compressed .dat files using a large Hash dictionary stored as a Memory-mapped file

This new approach has a number of advantages:

• The sort data volume is reduced as the compression is performed earlier in the process

• The sort can be optimised to work on our compressed file format so there is less overhead parsing input files

• The temporary disk space required during the build has been reduced

The following FastStats Designer features are supported

• Database and Delimited Data Sources

• Salesforce.com and Dynamic Database Data Sources

• All data types including Selectors, Text, Numeric (including Elapsed Time), Currency, References, Date and DateTime

• Flag Arrays

• Arrays

• The most commonly used mappings are supported (text mapping, postcodes, etc.)

• Orphan records are always loaded

• If incremental extracts are used then only the new records are appended before re-indexing to further improve performance

• Core & Linked Systems

• The following Lookup types:

  • SPL

  • SPL (merged)

  • MPL (join at query time)

  • Array/Flag array transformations

Enhanced Build Only Features

The following features are only supported by the enhanced build process and cannot be used by the conventional build process:

• Delimited Arrays
• Delta Updates
• Orphan Indicators

The following FastStats Designer features are not supported by the enhanced build process

• Fixed format and DBF data sources

• Only the most commonly used mappings are supported (postcodes, etc.)

• User defined Primary Selectors (auto generated is fine)

• Orphans cannot be excluded, they are always included

• The following Lookup types:

  • MPL (join at build time)

Is my design compatible with the Enhanced Build Process?

FastStats Designer will inform you if the Enhanced Build Process supports all the features required by your design. This is done after the 'View' (.vie) file is created at the start of the system build process (so after the Extract & AutoDiscovery Processes). The build process will fall back to using the conventional build process if a design feature is used that is unsupported however if a feature that is supported by the enhanced build process is required then the build will fail.